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(57)Abstract: 

PROBLEM TO BE SOLVED: To realize efficient image 
DB(database) management and utilization by applying a 
picture classification tree structuring method, an image 
classification presenting method, and picture retrieval 
integrating means for capturing whole image in an image 
DB. 

SOLUTION: Large classification points of views, such as 
an area color, a straight line, an object, and an image 
dictionary are presented from a system to a user for an 
image group in the middle of navigation. When a user 
selects the image dictionary, the system presents the 
classification points of view as *a human face', 'sky\ and 
Voof . When the user selects 'a human face', this system 
presents the set of portraits. In the same way, small 
classification points of view such as an area color, a 
straight line, an object, and an image dictionary are 
presented to the user for the image set, and the 
classification according to the selection of the 
classification points of view is conducted hierarchically. 
The history of the navigation is made valid for the classification of an image DB, and stored in 
this system as the classification tree information. The whole images of the stored classification 
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CLAIMS 



[Claim(s)] 

[Claim l] The hierarchical classification method of a picture of carrjdng out process in which show the 
classification result which made arbitrary picture sets applicable to a classification, and was classified 
according to the specified classification viewpoint, and the classification result further classified 
according to the classification viewpoint continuously specified to the aforementioned classification result 
is shown, as 1 time, and carrying out carrying out repeatedly two or more times as the feature. 
[Claim 2] The picture object which defined the concept equivalent to the language showing the concept of 
a picture, and this language. It has a field within a picture, and the table which matched tibe group of the 
conditions of the characteristic quantity of the straight line within a picture. The hierarchical 
classification method of the picture according to claim 1 characterized by extracting the information for 
classifying a pictiure and classifying a picture for the information which carried out [ aforementioned ] 
extraction as one of the classification viewpoints using the picture dictionary which connected these and 
the thesaurus of language. . 

[Claim 3] It asks for the color histogram of Y representation colors normalized with the area of the field 
which divided the picture and the picture. When there is a color in which the direction of a bright color 
has a threshold smaller than a dark color, and exceeds this threshold in the aforementioned color 
histogram It is the hierarchical classification method of the picture according to claim 1 characterized by 
extracting a color with the greatest value in the color as a representation field color of an object domain, 
considering as a field without a representation field color when there is no color exceeding this threshold, 
and classifying a picture by making into a classification viewpoint the representation field color which 
made [ aforementioned ] a decision. 

[Claim 4] the color histogram of Y representation colors normahzed with the area of the field which 
divided the picture and the picture asking the order from what has the aforementioned high 
histogram value - the the [ 1 main color and ] - the [ 2 main colors, ] - the hierarchical classification 
method of the picture according to claim 1 characterized by considering as Y main colors and classifying a 
pictxire by making these main colors into a classification viewpoint 

[Claim 5] It unifies to other sets with the hue which was similar in the picture set when there was less 
picture number of sheets belonging to a certain representation field color than the fixed number of sheets 
A, in case a picture was classified by making a representation field color into a classification viewpoint. 
Conversely, it is the hierarchical classification method of a picture the claims 1 or 3 characterized by 
dividing into the representation field color of a hue finer when there are more pictures belonging to the 
existing representation field color than the fixed number of sheets B (B>A), or given ia four. 
[Claim 6] The hierarchical classification method of the picture according to claim 1 characterized by 
classifying the aforementioned object picture into the sensibility word which performs distance 
calculation with the sensibility word characteristic quantity of the aforementioned sensibility word 
dictionary, and the color characteristic quantity extracted from an object picture, using the sensibility 
word dictionary which matched the characteristic quantity of the color imagined from a sensibihty word 
and its sensibility word as a classification viewpoint, and is most similar. 

[Claim 7] The hierarchical classification method of the picture according to claim 1 characterized by 
extracting the linear length, an angle, a number, and characteristic quantity called the relation between 
two or more straight Hnes from a picture, and making into a classification viewpoint characteristic 
quantity which carried out [ aforementioned ] extraction. 

[Claim 8] The physical relationship of the object within a pictxire extracted from a picture, its nimiber, 
and two or more objects within a picture, and the hierarchical classification method of the picture 
according to claim 1 characterized by making characteristic quantity called the distribution of the object 
within a picture into a classification viewpoint. 

[Claim 9] The hierarchical classification method of the picture according to claim 1 which makes a 



classification viewpoint the representation configuration beforehand registered as foundations, and is 
characterized by classifying the aforementioned picture into the representation configuration which 
extracts the amount of shape facilities of the object within a picture extracted firom 'the inside of the 
pictiu-e for a classification, and is most similar willi the distance calculation between those histograms. 
[Claim 10] It is the hierarchical classification method of a picture given [ to the claims 1-9 characterized 
by performing directions of a classification viewpoint by the navigation made to choose from from while 
showing combining two or more classification viewpoints ] in any 1 term, 

[Claim 11] It is the hierarchical classification method of the picture according to claim 10 carried out 
[ navigating a classification tree construction person by carrying out recommendation presentation, and ] 
as the feature in a user to order from a classification viewpoint with much number of sheets of the picture 
which corresponds to each classification viewpoint about the classification viewpoint for [ which removes 
the classification viewpoint which does not have the corresponding picture in case a classification 
viewpoint is shown from the candidate for presentation, and remains ] presentation. 
[Claim 12] The hierarchical classification method of the picture according to claim 10 or 11 characterized 
by holding selection of the classification viewpoint from the user in the process of the hierarchical 
classification of the picture by navigation as a classification history, and generating a classification tree 
from the aforementioned classification history. 

[Claim 13] The hierarchical classification method of a picture which is equipped with the following and 
characterized by classifying a picture, combining them freely; The hierarchical classification method of a 
picture given [ to the claims 3*9 which determine some examples of a typical classification item 
beforehand, and classify each picture into the nearest classification item when classifying a picture 
according to navigation hierarchical ] in any 1 term The clustering classification method of classifying the 
characteristic quantity of pictures, such as color characteristic quantity of each field, and the amount of 
shape facilities, into a set similar using the clustering technique 

[Claim 14] The classification / reference equipment of the picture carry out providing a history 
accumulation means hold directions of the classification viewpoint from the user in the process of 
navigation as a navigation' history, a picture classification tree generation means constitute a 
classification tree from the navigation history, a presentation means show the classification tree by which 
generation was carried out [ aforementioned ], and a reference means carry out the content retrieval of a 
picture by cooperating from the aforementioned classification tree when a referencie demand is made from 
a user as the feature. 

[Claim 15] In order that the presentation means of the aforementioned classification tree may use an 
electronic book and may express a classification result on this electronic book, the node of a classification 
tree is developed in the form of the chapter and paragraph of a book, for expression of the aforementioned 
node In order to express a chapter and a paragraph briefly, it expresses combining a classification 
viewpoint, synthetic drawing, arid a representation sample, and the outline of all the pictures included in 
the leaf is most displayed on the leaf of an end. the aforementioned reference means Classification / 
reference eqmpment of the picture according to claim 14 characterized by searching a picture by using the 
outline of the picture as a search key when the outline of arbitrary pictures is chosen from the 
aforementioned electronic book. 

[Claim 16] The record medium which recorded the program which performs the hierarchical classification 
method of the pictinre characterized by recording the program which performs procedure in the 
hierarchical classification method of a picture given [ to claims 1-13 ] in any 1 term by computer on the 
record medium which this computer can read. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] this invention employs efficiently multimedia systems 
which have a lot of pictures, such as a picture DB (database) system and an image DB system, and relates 
to the hierarchical classification method of the picture used in case it utilizes effectively, and equipment. 
[0002] 

[Description of the Prior Art] Until now, many picture managerial systems are realized. Two methods of 
reference and a classification mainly exist in the management method. 

[0003] # The method of searching using the text information (a keyword, explanatory note, etc.) added to 
the picture; and the method of searching with the degree of similar of the content information on a picture 
(characteristic quantity, such as a color and a texture) exist in picture reference picture reference. 
[0004] There are VIR of Informix and ExSight of NTT as a similar retrieval system using the content 
information on a picture. In ExSight, reference with the color of the whole picture, a pattern, the color of 
the object within a picture and a form, a position and the length of the straight line within a picture, an 
inchnation, and characteristic quantity like a position is enabled. 

[0005] # There are a method of classifying using the text information added to the pictxire, and a method 
of classifying using the content information on a picture also in the classification of a picture classification 
picture. 

[0006] As a classification system using the content information on a picture, the similar scene 
classification system of Netra of Calfornia size or the Kobe size exists, and they are clustered with the 
color of the block division field of the whole pictxure or a picture, or the characteristic quantity of a texture. 
[0007] # The navigation technique which supports narrowiug [ of a user's information ] down is proposed 
by showing automatically the viewpoint (keyword) which classifies the given set in the field of text 
navigation text reference further on the occasion of informational narrowing down: 
[0008] 

[The technical problem which invention tends to develop] The technical problem which should be solved 
[ exist / what has difficult grant of text information (pattern name) / like that it is difficult for cost and 
time to give the text iaformation to which the variation by this thing and - grant person and the whole DB 
cohered greatly since it cannot but depend for - grant on a handicraft, and - texture picture ] in grant of 
the text information on a picture existed. 

[0009] Then, although the technology of the content retrieval of a picture has been developed As an index 
and a table of contents are in a book, a general view of Picture DB cannot be grasped only by reference. - 
As the result The technical problem which should be solved [ be / difficult / to give a search key, though 
they are used, although a sketch input, a digital camera input, etc. exist as the acquisition method of that 
it becomes impossible to judge also about the quality of a reference result and - search key picture / it ] 
existed. 

[OOlO] Moreover, although the content classification technology of a pictxire is being developed in recent 
years -^Since it is the dedicated system to the picture of a special field which was developed imtil now like 
an aerial photograph picture or a cloud imagery and special characteristic quantity is used, are 
inapplicable to a general picture, • The technical problem which does not become the composition of a 
classification tree hierarchical only by clustering even when it can apply to a general pictxire, and does 
not function as a classification when the number of pictures increases after all and which should solve ** 
existed. 

[OOll] The technical problem of this invention is giving the picture classification tree building method for 
grasping the overview within Picture DB, the method of showing a picture classification, and the method 
of unifying a pictxire retrieval system, and reahzing the efficient method of picture DB managing / 
utilizing. 
[0012] 



[Means for Solving the Problem] this invention solves the above-mentioned technical problem by 
invention listed below. 

[0013] It is the hierarchical-classification method of a picture of carrying out the process which presents 
the classification result which the 1st invention made arbitrary picture sets applicable to a classification, 
and was classified according to the specified classification viewpoint, and is shown in the classification 
result further classified according to the classification viewpoint continuously specified to the 
aforementioned classification result as 1 time, and carrying out carrying out repeatedly two or more times 
as the feature. 

[0014] This invention is to the foundations of the foUowiag invention, and hierarchical classification of a 
picture and construction of a classification tree are realized by directions of a classification viewpoint, and 
the repeat of execution of the classification based on it. 

[0015] The picture object which defined the concept equivalent to the words with which the 2nd invention 
expresses the concept of a picture, and this language, It has a field within a picture, and the table which 
matched the group of the conditions of the characteristic quantity of the straight line within a picture. It 
is the hierarchical classification method of the picture characterized by extracting the information for 
classifyiog a picture and classifying a picture for the information which carried out [ aforementioned ] 
extraction as one of the classification viewpoints using the picture dictionary which connected these and 
the thesaurus of language. 

[0016] The thesa\irus of a text expresses the conceptual relation which "language" has. The "language" 
expressing a photographic subject can be expressed using picture characteristic quantity by defining the 
photographic subject which can express in the language reflected in a picture combining the feature of 
pictures, such as partial drawing, an object, and a straight line, between the pictures defined when using 
the conceptual relation of a keyword thesaurus here - relevance ****** - things are made, a picture 
dictionary is formed by summariziag these information, and it can use for extraction of a classification 
viewpoint 

[0017] The 3rd invention asks for the color histogram of Y representation colors normalized with the area 
of the field which divided the picture and the picture. When there is a color in which the direction of a 
bright color has a threshold smaller than a dark color, and exceeds this threshold in the aforementioned 
color histogram It is the hierarchical classification method of the picture characterized by extractiiig a 
color with the greatest value in. the color as a representation field color of an object domain, considering as 
a field without a representation field color when there is no color exceeding this threshold, and classifying 
a picture by making into a classification viewpoint the representation field color which made 
[ aforementioned ] a decision. 

[0018] For example, the histogram of the color in each field by which carried out subtractive color of the 
picture to the representation color of Y pieces, and X division was carried out in the same area is created. 
When the maximum of a histogram exceeds a specific threshold at this time, let the color equivalent to 
maximum be a representation field color in the field. When there is no color which fills the threshold, in 
the field, it is considered that there is no representation field color. In addition, the color per eye and at a 
cone's with the showy brighter color brighter than a dark and simple color sets a threshold to people's 
eyes low firom a dark color. The representation field color extracted by the above-mentioned method is 
used as a detailed classification viewpoint at the time of a user classifying, and the hierarchical 
classification based on this viewpoint is performed. 

[0019] the color histogram of Y representation colors which normaUzed the 4th invention with the area of 
the field which divided the picture and the picture asking the order firom what has the 
aforementioned high histogram value the - the [ 1 main color and ] -- the [ 2 main colors, ] - it is the 
hierarchical classification method of the picture characterized by considering as Y main colors and 
classifying a pictiu-e by making these main colors into a classification viewpoiat 

[0020] The case of a color typical about the whole picture where not one color but two or more domiaant 
colors not necessarily exist is almost the case, then, the order firom what of the whole pictxure field has a 
high color histogram value - the ■- the [ 1 main color and ] *• it considers as 2 main colors and This is 
extensible not only to the whole picture field but arbitrary picture fields. It is used as a classification 
viewpoint at the time of a user classifying, and the hierarchical classification based on the viewpoint is 
performed by tiiis result. 

[0021] It unifies to other sets with the hue which was similar in the pictiure set when there was less 
picture number of sheets belonging to a certain representation field color than the fixed number of sheets 
A, in case the 5th invention made a representation field color a classification viewpoint and a picture was 
classified. Conversely, when there are more pictures belonging to the existing representation field color 
than the fixed number of sheets B CB>A), it is the hierarchical classification method of the picture 
characterized by dividing into the representation field color of a finer hue. 



[0022] In the pictvure classification using the representation field color, when there is little number of 
sheets of the picture used as the candidate for a classification, or the picture number of sheets contained 
in one classification set by repeating a hierarchical classification has decreased gradually and it classifies 
using the representation color number specified beforehand, the classification set only whose several 
sheets include a picture may be formed. Such a classification makes the classification tree structure 
complicate unnecessarily. Then, when there is less picture number of sheets belonging to a certain 
representation color than the fixed number of sheets A, it unifies to other sets with the hue which was 
similar in the set, and the classification tree structure is simplified. Conversely, when there are many 
classification pictures, even if it classifies using the representation color fixed beforehand, one 
classification set may become very large. Then, when there are more pictxires with a certain hue than the 
fixed nimiber of sheets B after being once classified according to a representation color, additional offer of 
the specific classification viewpoint which can subdivide one classification set using the representation 
color in which hue division was finely carried out by the next classification is made. 

[0023] It is the hierarchical classification method of the pictru-e characterized by classifying the 
aforementioned object picture into the sensibiUty word which the 6th invention performs distance 
calculation with the sensibility word characteristic quantity of the aforementioned sensibility word 
dictionary, and the color characteristic quantity extracted from an object picture, using the sensibiUty 
word dictionary which matched the characteristic quantity of the color imagined from a sensibiUty word 
and its sensibility word as a classification viewpoint, and is most similar. 

[0024] In order to classify using "it is pretty" and the sensibiUty of "being fresh" which are received from a 
picture, the classification using the sensibiUty word dictionary which matched the characteristic quantity 
of the color imagined fi:om a sensibiUty word and its sensibility word is performed. Distance calculation 
with the sexual desire news extracted from a picture and the characteristic quantity which each 
sensibiUty word has is performed, and a picture is distributed and classified into a meaning to the 
sensibility word which was the shortest. 

[0025] It is the hierarchical classification method of the picture characterized by for the 7th invention 
extracting the Unear length, an angle, a number, and characteristic quantity called the relation between 
two or more straight Unes from a picture, and making a classification viewpoint characteristic quantity 
which carried out [ aforementioned ] extraction. 

[0026] The straight line extracted from the inside of a picture based on the information on an edge has 
many things reflecting the composition of a picture. However, with each straight-line information which 
consists of information on the starting point and a terminal point, it cannot use for a classification as 
information on the composition within the picture. Then, it investigates what relation **** straight lines 
have for whether it has the information in which the straight line extracted from the inside of a picture of 
one sheet reflects what composition using the relation between a linear number, length and a number, 
and two or more straight lines etc. again. It is used as a classification viewpoint at the time of a user 
classifying, and the hierarchical classification based on the viewpoint is performed by this result. 
[0027] Invention of the octavus is the physical relationship of the object within a picture extracted from a 
picture, its number, and two or more objects within a picture, and the hierarchical classification method 
of the pictTure characterized by making characteristic quantity called the distribution of the object within 
a picture into a classification viewpoint. 

[0028] The number of the object extracted from the inside of a picture and the distribution of the 
center-of-gravity position change with pictures, and this information also reflects the composition 
information on a picture. Then, the distribution situation which hit in one-sheet drawing many 
photographic subjects are reflected based on the number and the center-of- gravity position of an object is 
extracted. It is used as a classification viewpoint at the time of a user classif3dng, and the hierarchical 
classification based on the viewpoint is performed by this result. 

[0029] The 9th invention is the hierarchical classification method of the pictiu-e which makes a 
classification viewpoint the representation configuration beforehand registered as foundations, and is 
characterized by classifying the aforementioned picture into the representation configuration which 
extracts the amount of shape facilities of the object within a picture extracted from the inside of the 
picture for a classification, and is most similar with the distance calculation between those histograms. 
[0030] When classif)n.ng the picture set which has an object within 1 picture in 1 pict\ire like a clip art, a 
configuration serves as the very important feature. Then, some kinds of representation configurations 
which are to foundations, such as a round head, a square, and a rectangle, beforehand are set up, and it 
classifies into a most similar representation configuration at a meaning. SpecificaUy, distance calculation 
between the amounts of shape facilities of an object picture and a representation configuration is 
performed, and it classifies into the representation configuration which was the shortest. 
[0031] The 10th invention is the hierarchical classification method of the picture characterized by 



performing directions of a classification viewpoint by the navigation made to choose from from while 
showing two or more classification viewpoints. 

[0032] A classification is realized, while a user chooses the viewpoint of a field color, a straight line, an 
object, and a picture dictionary, in order to realize a hierarchical classification. Rough classification 
viewpoints (very kind viewpoint), such as a field color, a straight liae, an object, and a picture dictionary, 
are shown first. If a user chooses a classification viewpoint from this inside, a system will narrow down an 
applicable picture based on a previous selection viewpoint further, and will total based on a still finer 
selection condition (minor-key viewpoint). A user is shown the result as a selection condition with the 
picture sample applicable to each minor-key viewpoint. When a user chooses this minor-key viewpoint, a 
classification is actually performed and a classification set is created. A hierarchical classification is 
repeated navigating a user by presentation of such a classification viewpoint, and a picture is narrowed 
down. 

[0033] The classification viewpoint which does not have the corresponding picture in case the 11th 
invention presents a classification viewpoint is removed from the candidate for presentation, and it is the 
hierarchical classification method of a picture of carrying out navigating a classification tree construction 
person by carrying out recommendation presentation as the feature in a user to order, from a 
classification viewpoint with much number of sheets of the picture which corresponds to each 
classification viewpoint about the classification viewpoint for [ which remains ] presentation. 
[0034] In navigation, if there are many classification viewpoints too much, since a user does not 
understand in what may choose anything as a following classification viewpoint, an imnecessary 
classification viewpoint presents the candidate of a viewpoint who recommends while deleting it. Namely, 
when the classification by the field color, the straight line, and the object is performed to all the pictures 
stored in an image database, what corresponds to a classification viewpoint may not exist depending on 
the image data to apply. It is meaningless even if it shows such a result as classification selections. Then, 
such a classification viewpoint is shown to a user as a classification candidate who deletes from a 
classification selection candidate beforehand, and rearranges and recommends presentation sequence to 
order with many corresponding classification viewpoints. 

[0035] The 12th invention is the hierarchical classification method of the pictxire characterized by holding 
selection of the classification viewpoint from the user in the process of the hierarchical classification of 
the picture by navigation as a classification history, and generating a classification tree from the 
aforementioned classification history. 

[0036] The history of the classification viewpoint which the user chose at the time of the above-mentioned 
navigation is saved. The static classification tree which surveys the picture DB made into an object from 
this classification history is done. 

[0037] The hierarchical classification method of the above-mentioned picture of the 13th invention 
determining some examples of a typical classification item beforehand when classifying a picture 
according to navigation hierarchical, and classifyiag each picture into the nearest classification item, It is 
the hierarchical classification method of the picture characterized by having the clustering classification 
method of classifying the characteristic quantity of pictures, such as color characteristic quantity of each 
field, and the amount of shape facilities, into a set similar using the clustering technique, and classifying 
a picture, combining them freely. 

[0038] When classifying a miscellaneous picture variously, it is desirable for a classification to be possible 
using as many viewpoints as possible. Then, some examples of a tjrpical classification are beforehand 
decided like the hierarchical classification method of the picture of this inventions 3-9, and a user chooses 
and classifies two kinds, the "representation classification technique" which assigns each picture to the 
nearest item, and the "clustering classification technique" classified according to clustering characteristic 
quantity, such as color characteristic quantity of each field, and the amount of shape facihties, according 
to a situation. It becomes easy to form the set which is easy to carry out hierarchization of a classification 
by using such "compound classification technique", and is meaningful. 

[0039] A history accumulation means by which the 14th invention holds directions of the classification 
viewpoint from the user in the process of navigation as a navigation history, A picture classification tree 
generation means to constitute a classification tree from the navigation history, a presentation means to 
show the classification tree by which generation was carried out [ aforementioned ], and when a reference 
demand is made from a user It is classification / reference equipment of the picture characterized by 
providing a reference means to cooperate and to perform the content retrieval of a picture from the 
aforementioned classification tree. 

[0040] The history of the classification viewpoint which the user chose at the time of the above-mentioned 
navigation is saved. The static classification tree which surveys the picture DB made into an object from 
this classification history is done. Establish a functional means to peruse this static classification tree. 



and a hierarchy classification and similar reference are made to cooperate, and it becomes possible to 
narrow down a picture, changing reference to a classification according to a request of a user. 
[0041] In order that the presentation means of the aforementioned classification tree may use an 
electronic book and the 15th invention may express a classification result on this electronic book, the node 
of a classification tree is developed in the form of the chapter and paragraph of a book, for expression of 
the aforementioned node In order to express a chapter and a paragraph briefly, it expresses combining a 
classification viewpoint, synthetic drawing, and a representation sample, and the outline of all the 
pictures included in the leaf is most displayed on. the leaf of an end. the aforementioned reference means 
When the outline of arbitrary pictures is chosen from the aforementioned electronic book, it is 
classification / reference equipment of the picture characterized by searching a picture by using the 
outline of the picture as a search key. 

[0042] The node of the classification tree buUt as a result of using and classifying the form of the deep 
book of concordance every day as a means to provide a user with a classification result is expressed in the 
form of the chapter and paragraph of a book. In order to express each chapter and paragraph intelhgibly 
to the visitor of an electronic book, display representation drawing as node expression, a classification 
viewpoint is written by means of language, or synthetic drawing is displayed. When expressing the end 
leaf of a classification tree, all the pictures included in the leaf are displayed. In order to move to each 
chapter or a paragraph to those lower layer paragraphs, the link is stretched, by clicking the link, a lower 
layer paragraph can be displayed easily and a PEJIHE jump can be carried out. If arbitrary pictures are 
chosen firom the done electronic book, picture reference which used the picture as the search key can be 
performed. Moreover, if a favorite picture is chosen out of a reference result, the page by which the picture 
was carried with the electronic book will be opened. In case the above-mentioned node expression 
visualizes the hierarchical midst by navigation which is carrying out classification execution, and the 
classification tree structure of picture TOC to a user, it becomes possible [ expressing the node of a 
classification tree well ]. 

[0043] The 16th invention is the record medium which recorded the program which performs the 
hierarchical classification method of the picture characterized by recording the program which performs 
procedure in the hierarchical classification method of the above-mentioned picture by computer on the 
record medium which this computer can read. 

[0044] If this record medium is used, it will become possible to distribute the hierarchical classification 
method of the picture of this invention as a record medium, or to save it, and it will become possible to 
realize the method of this invention using a computer. 
[0045] 

[Embodiments of the Invention] Hereafter, the gestalt of operation of this invention is explained in detail 
using drawing. 

[0046] Drawing 1 shows the functional-block composition of the system which realizes the picture 
classification method by this invention, or equipment. The method of this invention is realized in each 
followiQg example of an operation gestalt, using these functional meanses suitably. The picture 
classification method by this invention is divided into (l) characteristic-quantity extraction phase, (2) 
classification tree construction phase, and (3) perusal phase. 

[0047] In a characteristic quantity extraction phase, while the characteristic quantity for reference is 
extracted firom each picture of the inputted pictiure set and being stored in the characteristic quantity DB 
for reference, each image data is stored in Picture DB. On the other hand, the characteristic quantity for 
a classification stated in the following examples of an operation gestalt from a picture by the 
characteristic quantity extraction means for a classification is extracted, and it is stored in the 
characteristic quantity DB for a classification. A picture dictionary and a sensibility word dictionary may 
be used in the case of extraction of this characteristic quantity for a classification. 

[0048] a classification -- timber -- he be carried out in a construction phase by repeating directions of the 
classification viewpoint from a construction person, and the classification of the picture by it by the 
classification tree construction support function, and a hierarchical classification does a classification 
tree is built from the classification history, and it is stored in the classification tree DB A classification 
tree construction support function is realized with a navigation means, a classification tree creation 
means, a clustering means, etc. 

[0049] In a perusal phase, a classification result (classification tree) is shown in this form by the browsing 
function by the presentation means. If a user emits a reference demand by this presentation, a picture 
reference means will search based on a classification tree, and will present a reference result with a 
presentation means. 

[0050] [Example 1 of an operation gestalt] The block diagram 2 of # picture dictionary (a), (b), and (c) are 
drawings explaining a picture dictionary. 



[0051] Drawing 2 (a) shows the example of the dictionary about empty. For example, a "blue sky" can be 
defined as the representation color of a pictxire object with a big area which exists in the upper part of a 
picture being **, or the representation color of the big field of the upper part of a picture is blue. A "cloudy 
sky" and a "night sky" can be defined similarly. Furthermore, the picture dictionary corresponding to the 
concept of "empty" can consist of combining with the thesaurus about empty. 

[0052] Drawing 2 (b) shows the example of the dictionary about "people's face." "People's face" is a round 
pictxire object and can be defined as a beige thing. 

[0053] Drawing 2 (c) shows the example of the dictionary about a "roof." Two straight hnes exist and a 
"roof can be defined as those physical relationship being mountain types. A picture dictionary like 
drawing 3 can consist of combination of the language showing these concepts, and its characteristic 
quantity, and combination of a thesaurus. 

[0054] [Example 2 of an operation gestalt] Extraction drawing 4 of # field color is drawing having shown 
the example of representation field color extraction of a picture. A metaphor is divided into seven patterns 
and 24 fields as shown in drawing. This serves as a classification viewpoint about a field. Furthermore 
the distribution about the color of each field is histogram-ized, and the maximum firequent appearance 
color which exceeded the specific threshold on it is determined as a representation field color of the field. 
When there is no color exceeding a threshold, suppose that he has no representation field color about the 
field. Calculation process of this representation field color is flow-chart-ized to drawing 5 . In addition, a 
bright color will be emphasized and the threshold will approach by human being's subjectivity, if you may 
differ for each color of every and the direction of a bright color has a threshold smaller than a dark color. 
[0055] The information on a field and its representation color is accumulated at the characteristic 
quantity DB for a classification. The picture in DB will be classified according to a field and its 
representation field color. In addition, a division field may also be a still finer field or a field which uses 
the diagonal line and is divided. 

[0056] [Example 3 of an operation gestalt] Extraction drawing 6 [ of # straight-Une-segments viewpoint ] 
(a) - (£) is the example of the classification viewpoint using the straight line. The characteristic quantity 
about the relation between the length of the straight line of each classification viewpoint, an angle, a 
number, and two or more straight lines is extracted for every picture or picture field, and it stores in the 
characteristic quantity DB for a classification. The flow chart of calculation of this straight-line- segments 
viewpoint is shown in drawing 7 . 

[0057] [Example 4 of an operation gestalt] # recommendation classification candidate's presentation 
drawing 8 shows the processing flow of the recommendation classification candidate presentation in a 
field color. After a representation field color is determined for every field to the arbitrary picture sets 
registered into the database, the number of sheets X of the picture which does not have a representation 
field color for every field is totaled. 

[0058] A user is shown the result into which this number of sheets X was rearranged in order of the few 
field as a recommendation classification candidate. A straight hne and an object can also be processed by 
the same method using a straight-line pattern or an object distribution pattern. 

[0059] By navigating a classification of a user by such recommendation candidate, classification tree 
construction work is done easy. 

[0060] [Example 5 of an operation gestalt] The flow chart is shown in the example of the navigation by 
presentation of a classification viewpoint, and drawing 10 at the navigation by presentation of # 
classification viewpoint, and use drawing 9 of a history. 

[0061] The candidate for reference is narrowed down in navigation, adding reference conditions (although 

it is generally narrowiag down, there is correction for reference etc. depending on the case). 

[0062] Furthermore, a classification viewpoint is shown in order to support next narrowing down like the 

conventional text navigation. However, the classification viewpoint serves as not a text (language) but a 

picture dictionary, a field color, a straight line within a pictvire, an object withia a picture, etc, 

[0063] In drawing 9 , further, to the picture set in the middle of navigation, the user was shown from the 

system the reference viewpoint of a field color, a straight line, an object, and a picture dictionary, the user 

chose the picture dictionary, and a system (or eqxiipment) presents "people's face", "empty", and a "roof*, 

and the user chose "people's face" and has got the set of a portrait. 

[0064] A user is shown the reference viewpoint of a field color, a straight line, an object, and a picture 
dictionary firom a system to the picture set still Uke the last step as a following step. Thus, one narrowing 
[ of a picture ] down is performed by choosing two kinds of viewpoints, rough classification viewpoints 
(very kind viewpoint), such as a picture dictionary, a field color, a straight hne within a pictxire, and an 
object within a picture, and the classification viewpoint (minor-key viewpoint) which detailed them. It is 
as the recommendation classification viewpoint presentation in the field color which is one of the kind 
viewpoints very much having been shown in drawing 9 . Moreover, about the minor-key viewpoint and 



the classification outline which a field color is shown very much as a kind viewpoint at the time of 
selection, the flow chart is shown in drawing 11 . 

[0065] In drawing 11 , the number of sheets Xi (0 <=i<=N) which belongs to each representation color (N 
color) to the field A inputted by the user is computed, it rearranges into order with much the number of 
sheets, a sainple is acqxiired out of registration image data in order with much number of sheets, and it 
shows with number of sheets. 

[0066] In the example of drawing 9 , as 2nd narrqwing down, the user chose the field color of No. 22, the 
system presented the classification outlines (statistical information about the classification etc.) of the 
field color of No. 22 further, and the user chose white and has got the set of the portrait of white dress. 
[0067] Since it tinrns out as a history of this navigation are [ of people ] a "face" and that "the field of No. 
22 is white'Vis effective in the classification of the picture DB present in use, the history is accumulated as 
static classification tree information to a system (or equipment). Picture DB is surveyable by perusing the 
overview of the acciimulated static classification tree using a keyword, synthetic drawing, and a sample 
picture. 

[0068] It cooperates a picture classification and similar reference like drawing 12 . In the process which 
narrows down the picture by classification, a key picture is inputted and similar reference is performed to 
acquire the picture to which a more concrete photographic subject is reflected out of the classification set 
at the time. For example, "the photograph with which both the persons and apples of white dress are 
reflected" can be acquired in inputting an "apple" as a key picture in a portrait set of the white dress of 
the classification set D of drawing 9 . Moreover, the change to the classification from reference is also 
enabled conversely. 

[0069] [Example 6 of an operation gestalt] # compound classification (fusion of a clustering classification . 
and a representation classification) 

When classifying a picture according to navigation hierarchical, some examjples of a typical classification 
item are decided beforehand, it has the clustering classification method of classifying into a set similar 
using the clustering technique the characteristic quantity of the hierarchical classification method of the 
picture of the above-mentioned example of an operation gestalt of classifying each picture into the nearest 
classification item, and pictures, such as color characteristic quantity of each field, and the amount of 
shape faciUties, and a picture is classified, combining them freely. 

[0070] When classifying a miscellaneous picture variously, it is desirable for a classification to be possible 
using as many viewpoints as possible. Then, some examples of a t3rpical classification are beforehand 
decided like the hierarchical classification method of an above-mentioned picture, and a user chooses and 
classifies two kinds, the "representation classification technique" which assigns each picture to the 
nearest item, and the "clustering classification technique" classified according to clustering characteristic 
quantity, such as color characteristic quantity of each field, and the amount of shape facihties, according 
to a situation. It becomes easy to form the set which is easy to carry out hierarchization of a classification 
by using such "compound classification technique", and is meaningful. It is shown as color clustering etc. 
into the classification viewpoint which sets clustering to one of the classification viewpoints, for example, 
is specifically shown to a picture set in drawing 9 . If this color clustering is chosen and directed, since 
clustering by color characteristic quantity will be performed and it will generally be classified into two or 
more picture sets, one of them is chosen and it shifts to the step of the next classification. 
[0071] [Example 7 of an operation gestalt] Degeneracy / fragmentation drawing 13 of # representation 
color is drawing explaining degeneracy of a representation color. 

[0072] It is decided using the technique of the example 2 of an operation gestalt that it will be either 
without the representation color to which the field color of a certain field was set beforehand or a 
representation color. For example, as shown in drawing, when you classify the picture of 1000 sheets into 
the representation color of 13 colors, suppose that there were only two pictures used as a light green color. 
It is better to constitute the simpleest possible classification tree, since [ which matte loan -comes to be 
hard ] the picture for which it asks where of the constituted classification tree is included, if the nimiber 
of leaves of a classification tree generally increases. Therefore, a certain threshold is prepared, and when 
only the picture below a threshold exists in one leaf, it unifies with a color similar to it. For example, like 
drawing 13 , only two sheets are unified with the near green of a hue, when a light green color does not 
exist. When the unified green set is picture number of sheets still fewer than a threshold, it combines 
with a blue classification further. A simple classification tree is formed by using such technique. 
[0073] When [ than the fixed number of sheets which, on the other hand, has the number of sheets of the 
picture classified into a certain color ] more, additional offer of the classification viewpoint which can 
divide the color in detail more is made as a following classification viewpoint. Drawing 14 shows the 
example of the detailed classification by color in the case of a representation color "blue." It enables it to 
classify the representation color "blue" used as criteria according to the color of several step story of "blue" 



to thin "deep blue." 

[0074] [example 8 of an operation gestalt] the [ # ] extraction drawing 15 of N main colors *• the -* it is 
drawing explaining extraction of N main colors 

[0075] Although representation classification by color of a whole field can be performed using the 
technique of the above-mentioned example 2 of an operation- gestalt, depending on a picture, as for a 
typical color, not one color but two or more dominant colors not necessarily exist in a whole field, then, 
drawing 15 *• like - the order from what of the whole picture field has a high color histogram value - the 
the [ 1 main color and ] it considers as 2 main colors and and the classification using these main 
colors is realized This is extensible not only to the whole picture field but arbitrary picture fields. 
[0076] [Example 9 of an operation gestalt] The example of a representation configuration is shown in # 
representation configuration classification drawing 16 . , and the flow chart of a representation 
configin-ation classification is shown in drawing 17 , 

[0077] In order to realize the classification based on the configuration, fundamental pictures, such as the 
round head and square like drawing 16 , and a rectangle, are prepared beforehand. These representation 
configurations can be changed to arbitrary pictures by the user according to the kind of picture used as 
the candidate for a classification. As shown in drawing 17 from the picture for a classification inputted as 
these representation configxirations, after performing distance calculation of a feature extraction and the 
amount of shape facihties, it classifies into the representation configviration which was the shortest. 
[0078] [Example 10 of an operation gestalt] The flow chart of a sensibiHty word classification is shown in 
# sensibility word classification drawing 18 . 

[0079] The system (or equipment) has the sensibility word dictionary which matched the color received 
from a sensibility word and its sensibility word. A user can choose the sensibility word registered 
arbitrarily. If the picture for a classification is inputted, the sensibility word characteristic quantity based 
on a color will be extracted, and distance calculation with aU the sensibility words registered into the 
sensibility word dictionary is performed, distance was the nearest in it - sensibiUty **** is carried out 
and an input picture is classified into a meaning 

[0080] [Example 11 of an operation gestalt] # electronic book system (or equipment) 
The example of an electronic book system (or equipment) is shown in drawing 19 . 

[0081] The node of the classification tree built by repeating a classification is expressed in the form of the 
chapter and paragraph of a book, and a table-of-contents page is prepared at the begiuning of an 
electronic book so that it may be easy to grasp the whole classification result. In order to express the 
content of each chapter and paragraph intelligibly briefly to the visitor of an electronic book, the 
explained representation drawing is displayed or a classification viewpoint is written by means of 
language. Only the end leaf of a classification tree creates the page which displays all the pictures 
included in the leaf. 

[0082] Like an actual book, the electronic book' done by the above-mentioned method can be turned over 
fluently, and can be perused. Moreover, in each chapter or a paragraph, from [ 1 ] can be jumped to the 
page which displays a lower layer paragraph easily as shown in (l) of drawing 19 by preparing the link 
function for moving to those lower layer paragraphs. 

[0083] When arbitrary pictiu'es are chosen from the done electronic book, reference which used the 
pictiu*e as the search key as shown in (2) of drawing 19 is performed. Moreover, if a favorite picture is 
chosen out of a reference result as shown in (3) of drawing 19 , the page by which the picture was carried 
with the electronic book will be opened. The above classifications -> you may try the seamless traffic with 
each technique of a reference and reference ■> classification how many times until the target picture is 
found. 

[0084] [Example 12 of an operation gestalt] The part or aU the functions of the functional-block 
composition shown by record-medium drawing 1 of the computer program which performs # hierarchical 
classification are realizable using a computer, Or it cannot be overemphasized that procedure shown by 
drawing 5 , drawing 7 , drawing 8 , drawing 10 , drawing 11 , drawing 12 , drawing 17 , drawing 18 , 
drawing 19 , etc. can be performed by computer. The program for a computer realizing the function of 
each part. Or the program for performing the procedure by computer It is possible to record the computer 
on the storage which can be read, for example, FD, (floppy disk), MO, ROM, memory card, CD and DVD, a 
removable disk, etc., to offer it, and to distribute it. 
[0085] 

[Effect of the Invention] According to this invention 1, a picture can be classified hierarchical and a 
classification tree can build easily. 

[0086] According to this invention 2, the hierarchical classification based on the photographic subject or 
concept of a picture is realizable. Moreover, a classification tree can be created in order of the 
classification near human being's classification feeUng. 



[0087] According to this invention 3, the hierarchical classification in the representation color near the 
impression of the color which people receive from a picture is attained. . « . , 

[0088] According to this invention 4, the classification of a picttire with two or more , dominant colors is 
attained. - . ■ 

[0089] According to this invention 5, the hierarchical representation field classification-bycolor result 
according to the scale of the aggregate is obtained. 

[0090] According to this invention 6, the hierarchical picture classification based on people's sensibility is 
attained. 

[0091] According to this invention 7, the hierarchical classification reflecting the linear composition of a 
picture is attained. ' ^ 

[0092] According to this invention 8, the hierarchical classification reflecting the composition by the object 
distribution of a picture is attained. 

[0093] According to this invention 9, the hierarchical classification by the similar configuration is 
attained. .... 
[0094] According to this invention 10, narrowing [ of the picture in process in which a picture is classified 
hierarchical ] down becomes easy. 

[0095] According to this invention 11, it can become possible to show a high order a promising 
classification viewpoint, among many classification viewpoints, and a user's bxirden can be made; to 
mitigate. . I * 

[0096] According to this invention 12, the classification tree of a picture can create easily. . . 

[0097] According to this invention 13, creation of the classification tree which the picture for which it asks 
tends to find is attained. < i * . > 

[0098] According to this invention 14, the classification tree of a picture can create easily, the overview of 
the picture registered into the database can be surveyed, and while grasping a general view of Picture DB 
can be searched. Thereby, discovery of the picture which had leaked from the. reference result-is also 
attained. . » , , 

[0099] According to this invention 15, a picture is discoverable, while a user can look through the 
classification tree of a pictxure easily and revolves a page with a clatter with an electronic book. Moreover, 
it can perform narrowing [ of a seamless picture ] down using both the functions of a classification and 
reference. 

[OlOO] According to this invention 16, the hierarchical classification method of a picture can be saved as a 
computer program, can be offered, or can be distributed, and hierarchical classification of a picture can be 
performed using a computer. 



[Translation done.] 



* NOTICES* 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. > ' 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are- not translated. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

; [Drawing l1 It is the functional-block block diagram of the system (or equipment) for realizing the 
hierarchical classification method of the picture by this invention. 

[Drawing 2l (a) is drawing in which the example of picture dictionary construction and (b) show the 
example of picture dictionary construction, and (c) shows the example of picture dictionary construction. 
[Drawing 3l It is drawing showing the example of composition of a picture dictionary. 
[Drawing 4l It is drawing showing the example of representation field color extraction of a picture. 
[Drawing 5l It is the calculation flow chart of a representation field color. 

[Drawing 6l (a), (b). (c), (d), (e), and (£) are drawings showing a straight-line-segments candidate's 
example, -^r ' , ; : • . . .. 

[Drawing 7l It is drawing showing the calculation flow chart of a straight-line- segments viewpoint. 
[Drawing 8l It is the flow chart of the recommendation classification viewpoint presentation in a field 
color. . . • 

[Drawing 9l It is drawing showing the example of the navigation classification using the classification 
viewpoint of Picture DB. 

[Drawing 101 It is the flow chart of the navigation by presentation of a classification viewpoint. 

[Drawing 111 It is the flow chart of the classification outline presentation in a field. 

[Drawing 12l It is the flow chart of a picture classification and picture reference cooperation processing. 

[Drawing 13l It is drawing showing the example of degeneracy of a representation color. 

[Drawing 14l It is drawing showing the detailed classification -by -color similar case (in the case of "blue") 

of a representation colori . ' 

[Drawing 15l the it is drawing explaining extraction of N main colors 

[Drawing 16l (a), (b), (c), (d), (e), CD, (g), and (h) are drawings showing the example of a representation 
configuration. 

[Drawing 17l It is, the flow chart of a representation configuration classification. 
[Drawing 181 It is the flow chart of a sensibility word classification. 

[Drawing 19l It is drawing explaining the example of an electronic book system (or equipment). 
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